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Appi. No. 09/833,934 

Response to Final Rejection dated March 15, 2004 
Reply to Office Action of September 24, 2003 

Applicants request reconsideration of the 
application in view of the remarks appearing below 
herein . 

Remarks 

Claims 1-10 remain in the application. 
I. THE ART REJECTIONS 

1. Claims 1, 3, 6 and 8 have been rejected under 35 
U.S.C. § 103(a) as being unpatentable over U.S, Patent 
6,240,217 Bl C^Ercan et al") in view of U,S. Patent 

6, 396, 539 Bl ("'Heller et al") . In support of the 
rejection, the examiner has generally asserted that steps 
(a) and (b) of present claim 1 are obvious since Ercan et 
al relates to digital image processing and, further, that 
although the primary reference does not explicitly 
specify recovering missing information. Heller et al 
teaches that compensation pixel values are generated to 
replace missing data from defective pixels. 

2. Claims 2 and 7 have been rejected under 35 
U.S.C. § 103(a) as being unpatentable over Ercan et al in 
view of United States Patent Application Publication No, 
US2002/0081019 Al ("Katayama et al") . In support of the 
rejection the examiner has generally asserted that 
although Ercan et al does not specify pattern-wise 
arrangement of image sensors, Katayama et al does teach 
providing an image sensing apparatus which determines 
whether the optimum image sensing conditions are achieved 
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or not by judging whether there is a predetermined 
pattern in an image sensing field* 

These art rejections are identical to those made in 
the first Office Action mailed April 9, 2003 in the 
application. Applicants continue to rely on the 
arguments made in their prior response filed July 9, 2003 
and these arguments will not be repeated here for the 
sake of brevity - 

The remarks here will be directed to the examiner' s 
comments which were directed to the arguments made by 
applicants in the previous response - 

A. The examiner has stated that Ercan et al . "... 
relates particularly but not exclusively to a method of 
selectively modifying pixel intensity information to 
improve legibility or visibility of parts of a digital 
image." Further, reference is made to Figs- 6A, 9A and 
lOA as illustrating missing information. 

Ercan et al relates exclusively to improving 
legibility or visibility of a digital image. Nowhere, 
does the reference even remotely suggest recovering 
missing color data in a two-dimensional color array. 
Fig. 6A illustrates an enhanced 4^bit image of the 
license plate of Fig. 3 as processed according to Example 
1; Fig. 9A illustrates the same license plate as 
■processed in Example 2; and Fig. lOA illustrates the same 
license plate as processed in Example 3. In none of 
these instances is missing information recovered. 
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It should be noted here that applicants, in the 
previous response, stated that ''Ercan et al relates to a 
black and white imaging method"- This statement is not 
correct. Nevertheless, the distinctions pointed out by 
applicants with respect to the method of Ercan et al. are 
correct . 

B. The examiner has stated that Ercan et al, in 
Fig. 5, illustrates a two-dimensional color array. This 
is not correct . The reference, at column 11, lines 9 - 
10 states "Fig- 5 illustrates the mapping arrangement 
used in the above [example 1] example". 

Subsequently, the examiner appears to have argued 
that example 5 of Ercan et al. illustrates a mapping 
arrangement wherein comparison of two images is carried 
out until the desired color image is obtained and that 
this process is the same as the method of applicants. 
Example 5 of this reference does not recover missing 
color data. 

C. The examiner has stated that Heller et al (Fig. 
6, step 84) illustrates that compensation pixel values 
are generated by examining the pixel values of the pixels 
surrounding the defective pixels. This may be so. 
However, it should also be noted that both embodiments of 
Heller et al use linear filters and there is no explicit 
teaching of the use of non-linear filters as are used in 
the method of applicants. Applicants' method involves the 
use of second color data and non-linear filters and the 
method of Heller et al does not- Applicants' method also 
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assumes that data is available at all pixels and 
therefore that defective pixels, such as are addressed by 
Heller et al do not exist or have been previously- 
compensated. 

D. The examiner has stated that it is the 
examiner' s interpretation that "...correcting defective 
pixels on image sensing elements would have the same 
concept as correcting missing information pixels on image 
elements." It is submitted that this statement does not 
relate to the teaching of Heller et al. 

The issue to be addressed is whether the rejection 
of the claims is proper based on the teaching (s) of the 
reference (s) - Here it has been shown that Heller et al 
does not teach or suggest recovering missing color 
information. 

E. The examiner has referred to the fact that 
Ercan et al refers, in the Abstract, to using a non- 
linear algorithm. This teaching is in the context of the 
overall method taught by the reference. As pointed out 
in detail, this reference does not teach recovering 
missing color information and the examiner has 
acknowledged this fact. 

There is no suggestion in Ercan et al which would 
provide any incentive for one skilled in the art to take 
the reference to a non-linear algorithm out of the 
context of the overall teaching and incorporate a non- 
linear algorithm into the method of Heller et al. It is 
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impermissible to take from one reference only so much as 
is required to combine with another reference to arrive 
at the claimed invention. There must be some suggestion 
in the reference to do so. Here, since the respective 
methods of Ercan et al and Heller et al are directed to 
different results, there would be no suggestion to use a 
non-linear filter in the method of Heller et al. In 
fact, if a non-linear filter were to be used in the 
method of Heller et al the intended method would not be 
operative for its intended purpose . 

In any event, even if, assuming arguendo, Ercan et 
al and Heller et al were to be combined in the manner 
suggested by the examiner, the combined disclosures do 
not amount to applicants' method* As has been discussed 
in detail, neither of the references involves the 
application of two one-dimensional non-linear 
interpolation processes . 

F. With respect to claims 2 and 7, the examiner 
has stated that '^Katayama on page 3 paragraph 0082, in a 
second embodiment teach correcting missing color on the 
image, and also see Fig. 31." 

Paragraph 0082 of this reference states, with 
respect to a second embodiment that ^\..it is characterized 
by correcting a three-dimensional image on the basis of 
predetermined image sensing parameters or editing a 
three-dimensional image.". Nothing in this statement or 
the illustration shown in Fig. 31 deals with applicants' 
claimed method and apparatus directed to recovering 
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missing color data in a two-dimensional color array 
involving the application of two one-dimensional non- 
linear interpolation processes. 

As discussed previously during prosecution, Katayama 
et al is directed to image sensing apparatus capable of 
placing an object, whose three dimensional shape is to be 
generated, under the optimum image sensing conditions 
upon sensing the object from a plurality of image sensing 
points. This reference is not directed to recovering 
missing color information. 

In summary, applicants' claimed method and apparatus 
are directed to recovering missing color data in a two- 
dimensional color array. The method and apparatus 
involve the application of two one-dimensional non-linear 
interpolation processes* Generally speaking, applicants' 
method essentially "decouples" the conventional two- 
dimensional process into two one-dimensional processes* 
The first one-dimensional color recovery process 
generates intermediate second color image data from the 
sampled first color image data and the second one- 
dimensional color recovery process generates the desired 
third color image data from the second color image data. 

As described in detail in the application (see, for 
example, page 8, line 28 to page 10, line 14) a two- 
dimensional array of discrete image sensing elements, 
each of which is specifically responsive to one of at 
least three predetermined colors (e.g., red, green and 
blue) is exposed to image information-bearing 
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illumination to obtain a collection of each electronic 
information signal received from each discrete element. 
The collection of signals forms the raw unprocessed one- 
color image data from which fully recovered third color 
image data can be derived. 

The first step towards deriving the fully-recovered 
third color image data is to first recover missing color 
information along a first dimension (e.g., along rows of 
the array) by interpolating the first color image data 
along the first dimension to provide a first- 
interpolated color data for each of the discrete 
elements, forming a difference channel between the first 
color image data and the first interpolated color data 
and then applying a one-dimensional non-linear filter on 
the difference channel and combining with the first color 
data to obtain the first recovered image data. The second 
color image data comprises a combination of the first 
recovered image data and the first color image data. 

The second step of the method derives fully- 
recovered third color image data from the two color image 
data by recovering missing color information along a 
second dimension, e.g*, along columns of the- array* The 
second color image data is obtained by interpolating 
along the second dimension to provide second interpolated 
data for each of the discrete elements, forming a 
difference channel between the second color image data 
and the second interpolated data and then applying a one 
dimensional non-linear filter on the difference channel 
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and combining with the second color data to obtain the 
second recovered image data- The third color image data • 
comprises a combination of the second recovered image 
data and the second image data. 

Here it is readily apparent that the teachings of 
the references cannot be combined to arrive at the 
claimed method and apparatus of applicants and therefore 
the disclosures of the references do not properly support 
the art rejections applied against claims 1, 3, 6 and 8. 

Reconsideration of the art rejections applied to 
claims 1 -3 and 6-8, and withdrawal thereof are 
respectfully requested. 

II. THE SECTION 112 REJECTIONS 

Claims 3-5 and 8-10 have been rejected again 
under the first paragraph of 35 U.S.C. § 112 as 
containing subject matter which was not described in the 
specification in such a way as to reasonably convey to 
one skilled in the relevant art that the inventors, at 
the time the application was filed, had possession of the 
claimed invention . 

A. The examiner has stated that claims 3 and 8 
should specify the specifications and parameters for 
rank-order filters. It is further stated that it would 
be useful to overcome the rejection, for applicants to 
provide the assumptions for designing the non-linear 
filter, the formulation for the filter and the 
limitations of the filter. 
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The specification teaches that the filter functions 
to remove narrow peaks and valleys within the color 
difference data. See, for example, page 21, lines 26- 
28. It is also taught that a preferred non-linear filter 
is a median rank order filter (see, for example, page 14, 
lines 25-27). It is specified that a suitable rank 
order filter for use according to the claimed method and 
apparatus is described in United States Patent No. 
4,802,108 which is incorporated by reference in the 
present application . 

The function of a rank order filter on a set of 
values is to arrange the values in rank order, from 
smallest to largest, and then to select the value that 
occupies a specified rank in that order. For example, 
the ^'minimum" function is a rank order filter that 
selects the value that is the smallest (lowest in rank) ; 
the so-called ^"median" function is a rank order filter 
that selects the value of middle rank. For example, the 
median of the five numbers 1, 1, 5, 7, 2 is found by 
arranging the numbers in rank order, that is, 1, 1, 2, 5, 
7 and then selecting the middle value which is 2. 

It is known that rank order filters are valuable for 
removing localized impulses, or deviations, in a data 
stream. For example, the data sequence 0012110 15 0 
121011 is a stream with numbers typically between 0 
and 2 but with an outlying value of 15 within it. If a 
median filter of length 3 were to be applied to this data 
stream the following results would be obtained 
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Median 



First 3 values 1, 2, 1 1 

2, 1, 1 1 

1, 1, 0 1 

1, 0, 15 1 

0, 15, 0 0 

15, 0, 1 1 

0, 1, 2 1 

1, 2, 1 1 

2, 1, 0 1 
1, 0, 1 1 



and so forth* 

The median filter has thus eliminated the outlying 
value. In a group of three numbers, the number 15 was 
always the largest and never the mean so it was never 
selected by the filter. In general, when the purpose of 
the rank order filter is to remove outlying values in a 
data stream,, a median filter is a preferred choice since 
the outlying value will generally be the largest or the 
smallest value in any ranking and therefore will not be 
selected by the median filter. 

The design parameters of rank order filters are: 
(a) the '^length" of the filter, (i.e,, the number of 
values for which the ranking is done) ; and (b) the rank 
order to be selected (i.e., lowest, middle, etc.). These 
parameters are tailored to the specific application. For 
example, in situations in which isolated outlying values 
are expected, a median filter of length 3 or greater may 
be selected because in this case each outlying value will 
be ranked along with at least 2 "normal" values and one 
of the normal values will be selected as the median. 
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When outlying values may occur in pairs^ as may be the 
case in some of the color difference signals examples in 
the present application, then a median filter of length 5 
or greater may be used. In this case, the two outlying 
values will always be ranked along with at least 3 
''normal" values and one of the 3 normal values will be 
selected as the median. 

The design considerations for rank order filters are 
known in the art and are taught, for example, in U.S. 
Patent 4,663,655 which has been cited as prior art in 
this application. The operation of rank order filtering 
may be exercised in software or hardware as described in 
U.S. Patent 4,802,108 {incorporated by reference herein) 
and the references cited therein. 

Thus, in view of the teaching of the present 
specification, together with the state of the art with 
respect to rank order filters, as illustrated by the 
foregoing remarks and the cited patents and the journal 
articles submitted by applicants with their previous 
response, it is submitted that applicants clearly were in 
possession of the subject matter of claims 3 and 8 at the 
time the application was filed, 

B- The examiner, regarding claims 4, 5, 9 and 10, 
has asked ''How does Applicant consider a combination of 
red and green wavelengths, while there are other colors 
such as orange and yellow colors between the red and 
green wavelengths?" 
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The application teaches that three different 
predetermined colors are used and each discrete element 
is responsive to one of three different predetermined 
colors. Claims 4 and 9 recite that the three 
predetermined colors are a color substantially within the 
red wavelengths, a color substantially within the green 
wavelengths and a color substantially within the blue 
wavelengths. Claims 5 and 10 recite that the 
predetermined colors are a color substantially within a 
combination of red and green wavelengths, a color 
substantially within a combination of green and blue 
wavelengths and a color substantially the red and blue 
wavelengths , 

Generally speaking, red wavelengths are in the 
vicinity of 650 nm, green wavelengths are in the vicinity 
of 550 nm and blue wavelengths are in the vicinity of 450 
nm. 

In the case of CCD imagers as utilized in the 
present invention, it is preferred to record color by 
placing a red, a green and a blue color filter over each 
pixel and to have a fraction of the pixels responsive to 
each of these primary colors. Thus, each pixel is 
responsive to a single one of the three primary colors. 

It should be recognized that the present claims 
require the discrete elements to be responsive to the 
recited wavelengths. Each discrete element may be 
responsive to other wavelengths also. Thus, a discrete 
element may be responsive to wavelengths in the red 
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region and also to wavelengths in the orange and/or 
yellow regions - 

Those skilled in the art are clearly familiar with 
the color principles embodied in the claimed subject ■ 
matter. That there are orange and yellow colors in the 
spectrum between the red and green wavelengths does not 
render claims 4, 5, 9 and 10 defective under the first 
paragraph of Section 112. All that is required for the 
operation of the method is that" each discrete element be 
responsive to one of three predetermined colors. 

In view of the foregoing, it is submitted that 
applicants clearly were in possession of the subject 
matter recited in claims 3-5 and 7 - 10 at the time the 
application was filed. Accordingly, these claims comply 
with the criteria for patentability established by the 
first paragraph of 35 U.S.C. § 112. 

Reconsideration of these grounds of rejection and 
withdrawal thereof are respectfully requested. 



Polaroid Corporation 
Patent Department 
1265 Main Street 
Waltham, MA 02451 
Tel. : 781-386-6405 
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Respectfully submitted. 




Gaetano D. Maccarone 
Registration No. 25,173 
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